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Intervention Cognitive Tutor®, a full-year course, covers organizing single variable data, simplifying linear expressions, mathematical modeling, 
solving systems with linear equations, problem-solving using proportional reasoning, and powers and exponents. Students work at 
their own pace to develop problem-solving skills. The duration of each lesson can vary, depending on the length of a school’s class 
period. Generally, three periods a week are for classroom activities using the Cognitive Tutor® text, and two are spent in the computer 
lab using the Cognitive Tutor® software. 



For Middle school students. 



Findings 



Evidence base 



Evidence limits 



In one randomized controlled trial, students in the Cognitive Tutor® group earned higher average scores on the Educational Testing 
Service Algebra I test and higher end of semester grades than did the comparison students in algebra classes that used McDougal 
Littell’s Heath Algebra I curriculum. 

^ 1 randomized controlled trial meets evidence standards. 

J 0 quasi-experimental design studies meet evidence standards with reservations. 

Q 5 studies do not meet evidence screens. (see symbol key on page 6) 

The evidence base for Cognitive Tutor® is limited to one randomized controlled trial of 369 9th graders attending urban middle schools. 



Scope of use 



Developer and contact 



Cognitive Tutor® was first implemented in 1 992. As of September 2004, it was used in about 1 ,500 schools by about 1 70,000 
students. 

Carnegie Learning Inc., www.carnegielearning.com ; email: info@carneqielearninq.com; telephone: (888) 851-7094. 
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Cognitive Tutor®, developed by Carnegie Learning Inc., includes a 
set of curricula for Algebra I, Geometry, Algebra II, Integrated Math 
I, II & III, and Quantitative Literacy Through Algebra. This report 
focuses on one of its products used in middle schools: Cognitive 
Tutor® Algebra I. 

Cognitive Tutor® is designed to help teach algebra with 
interactive computer software in personalized computer 
sessions. The goal of the curriculum is to improve students’ 
understanding of basic foundational skills as well as high-order 
mathematical concepts. The curriculum covers the following 
eight topics: 

• Organizing single variable data 

• Simplifying linear expressions 

• Finding linear equations from graphs 

• Mathematical modeling 

• Solving systems of linear equations algebraically and graphically 

• Solving and graphing equations involving absolute values 

• Problem-solving using proportional reasoning 

• Powers and exponents 

The developer states that these topics are designed to align 
with the National Council of Teachers of Mathematics (NCTM) 
Curriculum and Evaluation Standards (Corbett, Koedinger, & 
Hadley, 2001). 

Cognitive Tutor® course materials for students include a 
textbook, software, and supplemental materials. Materials for the 
teachers consist of a teacher’s edition of the textbook, student 
assignments with answers, assessments, a software manual, and 
a teacher’s toolkit for student engagement activities. 

Cognitive Tutor® is a full-year course with each lesson lasting 
the full class period. 

Teaching 

In the classroom, teachers facilitate small group problem-solving 
and whole classroom discussions. In the computer lab, teachers 
interact with students on a more individual basis, and Cognitive 
Tutor® acts as a classroom assistant to facilitate student progress. 

Carnegie Learning provides a three-day preservice training. 
In-service professional development is also available during the 



year. Teacher training for all Cognitive Tutor® curricula provides 
educators the opportunity to understand the philosophy and 
application of these products. The training sessions are 
conducted by Certified Implementation Specialists, each a 
current or former mathematics teacher who has completed 
in-depth training from Carnegie Learning’s staff of educators, 
technology specialists, and curriculum developers. 

Typical lesson 

Three class periods a week are organized around textbook 
materials and small group activities. In the other two class 
periods, students work at their own pace to develop their own 
problem-solving skills by working with the Cognitive Tutor® on 
the computer. 

The software uses problem scenarios that pose multiple 
questions to the students. In some problems, the software 
presents a problem scenario for which the student must create a 
mathematical model, using equations, graphs, and tables. 
Students are expected to make connections between the 
symbolic, graphical, and tabular representations and the written 
problem — and provide justifications for the methods they use. In 
other units, students are expected to solve more abstract 
mathematics problems, such as linear equations, again justifying 
their approach and reasoning at each step. During this process 
students receive immediate feedback when they make errors. 

Scope of use 

Pilot implementation of the curriculum began in 1992 with 84 
students in one school. As of 2004, it is in use in about 1 ,500 
U.S. schools by about 170,000 students. 

Cost 

According to the 2004/05 pricing information, the price for 
Cognitive Tutor® ranges from $716 for a five-user lab pack to 
$12,500 for an annual site license for 400 or more users. 

Regional training is included in the site license, and schools 
buying lab packs can opt for onsite training for an additional 
$2,500. 
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Study findings 



Strength of the 
evidence base 



References 



WWC Intervention Reports 



Randomized controlled trial significantly higher mean scores on the Educational Testing 

In the single study on the Cognitive Tutor® that meets evidence Service Algebra I test and on their end-of-semester grades than 
standards, students in the Cognitive Tutor® group earned did the students in the comparison group. 



The WWC collected more than 800 studies for the Middle School 
Math Curriculum review. Five studies looked at Cognitive Tutor®. 
Four failed to meet WWC evidence screens. All four of the studies 
were quasi-experimental design studies that did not account for 
pre-existing differences between groups with matching or equating. 

Studies were rated according to the strength of their causal 
evidence. Studies that placed students into the intervention and 
comparison groups randomly (randomized controlled trials) without 
notable design or implementation flaws are classified as meeting 
evidence standards (© a ). Other studies that use comparison 
groups (quasi-experimental designs) and randomized controlled 
trials with notable flaws are classified as meeting evidence 
standards with reservations ( ✓ a ). 

One study on Cognitive Tutor® meets WWC evidence 
standards. WWC focused on the part of this study that randomly 
assigned students to groups. 



Studies are further rated for intervention fidelity, outcome 
measures, breadth of evidence, reporting on subgroups, analysis, 
and statistical reporting. That information is provided in study 
reports, but does not affect the overall rating. 

The intervention in this study is well defined and has no noted 
implementation problems. The primary outcome measures, ETS 
Algebra I Test and end of semester grades, appear to be valid 
and aligned with the intervention. Appropriate methods were 
used to estimate average effects of the intervention. The study 
looked at and reported findings for only 9th-grade students in an 
urban Oklahoma district. 

Table A2 describes the one outcome study of Cognitive Tutor® 
that meets WWC evidence standards. For a more detailed 
description of the study, see the Detailed Study Report or Brief 
Study Report. 

a See symbol key on page 6. 
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Table 1 

Effects 



Study 



Measure 



Favors 

comparison group 
-2.00 -1.50 



- 1.00 



-0.50 



Comparison 
group mean 

0 0.50 



1.00 



Favors 

Cognitive Tutor® group 
1.50 2.00 



© Morgan Educational Testing 
& Ritter Service Algebra Test 
2002 (N=360 students) 












0.23 










Semester 1 grade 
(N=369 students) 










h 


0.17 










Semester 2 grade 
(N=364 students) 












1 










Approximate percentile ranking 


2% 7% 16% 31% 50% 69% 84% 93% 98% 



How to read this table: The wide, shaded bar indicates both the direction and estimated size of the effect of the intervention. The estimated effects reported here are 
standardized differences in the mean values between the intervention and comparison groups. Bars extending to the right of zero denote estimated effects that favor the 
intervention group, and those extending to the left of zero denote estimated effects that favor the comparison group. The solid line through the shaded bar marks the 95% 
confidence interval of the estimated effect; when there is no solid line, the study did not provide data to correctly compute the confidence interval. When the line does not cross 
zero, the estimate is statistically significant. The bar is striped if the effect is not statistically significant or if significance could not be accurately computed. The scale at the 
bottom of the chart indicates the approximate percentile distribution of students in the comparison group. The percentile ranking at the end of the shaded bar can be used to 
interpret the standardized mean difference in the outcome. For example, an effect of 0.5 is roughly equivalent to an increase in the mean value from that of the average 
student in the comparison group (50th percentile) to that of the average student in the 69th percentile. 



What Works Clearinghouse 



The What Works Clearinghouse (www.whatworks.ed.gov) was established in 2002 by the U.S. Department of Education’s Institute of Education 
Sciences to provide educators, policymakers, researchers, and the public with a central and trusted source of scientific evidence of what works in 



education. Please email all questions and comments to info@whatworks.ed.gov. The What Works Clearinghouse is administered by the U.S. 
Department of Education through a contract to a joint venture of the American Institutes for Research and the Campbell Collaboration. 




Appendix 



Table ai Summary characteristics and findings from randomized controlled trials on Cognitive Tutor® 











Sample size 




Mean outcome 


Standard deviation 3 


Estimated impact 11 


Study 


Study 

sample 


Measure 


Intervention Comparison 
group group 


Total 


Intervention 

group 


Comparison 

group 


Intervention 

group 


Comparison 

group 


Mean 

difference 


Standardized 
mean difference 


Morgan 
& Ritter 
2002 


9th-grade, 
regular education 
students 


Educational 
Testing Service 
Algebra Test c 


222 


138 


360 

students 


16.7 


15.4 


5.7 


5.6 


1.3 


0.23 (±0.21) 






Semester 1 
grade 


190 


179 


369 

students 


2.9 


2.7 


1.1 


1.2 


0.2 


0.17 (±0.20) 






Semester 2 
grade 


187 


177 


364 

students 


2.7 


2.4 


1.2 


1.3 


0.3 


0.24 (±0.21) 



a Shows how dispersed the participants' outcomes are. A small standard deviation would suggest that participants had similar outcomes, 
b The WWC estimated impact based on statistics reported by the study author. 

c This is not a state or nationally normed standardized test, but it appears to be valid and aligned with the intervention. 
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Table A 2 Characteristics of interventions in reviewed studies on Cognitive Tutor®: Morgan & Ritter 2002 



Evidence 

base 

rating 


Characteristic 


Description 


0 


Study citation 


Morgan, P., & Ritter, S. (2002). An experimental study of the effects of Cognitive Tutor Algebra 1 on student knowledge 
and attitude. (Available from Carnegie Learning, Inc., 1200 Penn Avenue, Suite 150, Pittsburgh, PA 15222) 




Participants 


369 9th-grade students in the part of the study that used random assignment. About two-thirds of the students in the 
study were white, with the other third Asian, black, Hispanic, Native American, and other. None of the students was in 
special education. 




Setting 


Four urban junior high schools in the Moore Independent School District in Oklahoma (in the part of the study that 
used random assignment). Six teachers in three of four intervention schools taught both the intervention and the 
control curricula. 




Intervention 


The intervention group was taught Algebra 1 using the Cognitive Tutor® Algebra 1. The intervention is designed to be 
a full-year course. Students in this group spent three class periods each week in group activities and classroom 
discussions and two developing their own problem-solving skills at their own pace by working with the Cognitive 
Tutor®. 




Comparison 


The comparison group was taught Algebra 1 using the McDougal Littell’s Heath Algebra /, a traditional, 
teacher-directed curriculum. The study does not provide further information on this curriculum. 




Primary outcomes and 
measurement 


ETS Algebra 1 End-of-Course Assessment, developed by ETS, consisting of 25 multiple-choice and 15 
constructed-response questions. The other two outcomes were semester 1 grades and semester 2 grades. 



Symbol key 
for evidence 
base rating 



0 Study meets evidence standards (randomized controlled trial without notable flaws). 

/ Study meets evidence standards with reservations (randomized controlled trial with notable flaws or quasi-experimental 
design study without notable flaws). 



© 



Study does not meet evidence screens. 
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